Sphericity 
and multiplication of double cosets 
for infinite-dimensional classical groups 

Yury A. NeretiisQ 

We construct spherical subgroups in infinite-dimensional classical groups G (usu- 
ally they are not symmetric and their finite-dimensional analogs are not spherical) . We 
present a structure of a semigroup on double cosets L\G/L for various subgroups L 
in G, moreover these semigroups act in spaces of L-fixed vectors in unitary representa- 
tions of G. We also obtain semigroup envelops of groups G generalizing constructions 
of operator colligations. 

1 Introduction 

1.1. Definition of spherical subgroups. Let G be a group, K C G a 
subgroup. An irreducible unitary representation p of G in Hilbert space H is 
called K-spherical, if there exists a unique (up to a proportionality) i\-fixed 
vector v G H. The spherical function of a spherical representation is given by 
the formula 

Hg) = (p(g)v,v). 

Let G be a semisimple Lie group, K a compact subgroup. The pair (G, K) is 
called spherical, if any irreducible unitary representation of G contains at most 
one non-zero if-fixed vector. The subgroup K is said to be a spherical subgroup 
of G. 

According to the Gelfand theorem jj, symmetric subgroups are spherical. 
In particular, maximal compact subgroups of semisimple groups are spherical. 

Kramer classified [TU] all spherical pairs (G, K) with simple group G, the 
classification was extended to semisimple groups in [T3] and [2J. All such pairs 
can be produced from symmetric pairs (G, K) by a minor diminishing of the sub- 
group K or small enlargement of G. On spherical functions for non-symmetric 
pairs, see the work of Knop [8]. 

1.2. Infinite-dimensional classical groups. We consider Hilbert spaces 
I2 over M, C, or quaternions EL We call a matrix g, whose elements are con- 
tained in R, C, H, finite if the matrix g — 1 has only finite number of non- 
zero elements. We define infinite- dimensional classical groups U(oo), O(oo), 
GL(oo,M), Sp(2oo,M), U(oo,oo), ...as groups of finite matrices satisfying the 
usual identities. All such groups G(oo) are inductive limits of corresponding 
finite-dimensional groups G(n), 

Sp(2oo,M) = limSp(2n,R), U(oo, 00) = limU(m, n 2 ), 
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We suppose that G(n) are canonically embedded to G(oo) and act to initial basis 
vectors. If we consider groups of type O(oo, oo) etc. or products of groups, then 
we regard n in G(n) as an integer vector n = (m, . . . ). 

1.3. Heavy groups. The orthogonal group O(oo), the unitary group 
U(oo), and the quaternionic unitary group Sp(oo) play the same role as compact 
groups in finite-dimensional theory. We call them heavy. For a heavy group K , 
we denote by K a the stabilizer of initial a basis elements in the corresponding 
1%. We write elements of K a as block (a + oo) x (a + oo) matrices of the form 



Let p be a unitary representation of a heavy group in a Hilbert space H. 
Denote by ffW the space of i^ Q -fixed vectors. 

Theorem 1.1 [23j The following conditions are equivalent: 

— p is continuous with respect to the weak topology on K ; 

— p is continuous with respect to the uniform topology on K ; 

— U-ff [ Q l is dense in H ; 

— p admits an extension to a weakly continuous representation of the semi- 
group of operators with norm ^ 1 in £jj. 

If p is irreducible, this is equivalent to 
— _ff[ a l ^ for some a. 

We call such representations continuous. Description of continuous represen- 
tations is contained in [23j . all such representations can be obtained by tensor 
operations from the simplest representation of K in £ 2 - 

Finite products of heavy groups also will be called heavy. If K = Ki x 
• • • x K p , then we assume K a := K" 1 x • • • x Kp p , now a — (ai, . . . , a p ) is a 
multi-index. Irreducible representations of a heavy group K — K\ x • • • x K p 
are tensor products of representations of Kj . 

1.4. Olshanski's list. The following list consists of pairs (G, K)= (classical 
group, symmetric heavy subgroup). Explicits form of embeddings K(oo) — > 

2 We call such operators contractions. 




, where h G K. 
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G(oo) are sufficiently obvious. 

Pairs of noncompact type Pairs of compact type 

(GL(oo,M),0(oo)) (U(oo).O(oo)) 

(GL(oo, C), U(oo)) (U(oo) x U(oo), U(oo)) 

(GL(oo,H),S P (oo)) (U(2oo),Sp(oo)) 

(U(oo, oo), U(oo) x U(oo)) (U(2oo), U(oo) x U(oo)) 

(Sp(2oo,R),U(oo)) (Sp(oo),U(oo)) 

(Sp(2oo, C), Sp(oo)) (Sp(oo) x Sp(oo), Sp(oo)) 

(Sp(oo, oo), Sp(oo) x Sp(oo)) (Sp(2oo), Sp(oo) x Sp(oo)) 

(O(oo, oo ), O(oo) x O(oo)) (O(2oo), O(oo) x O(oo)) 

(0(oo,C),0(oo),) (O(oo) x O(oo),0(oo)) 

(SO*(2oo),U(oo)) (0(2oo),U(oo)) 

Also there are 3 series of symmetric pairs of finite rank: 

(U(p, oo), U(p) x U(oo)) (U(p + oo), U(p) x U(oo)) 

(00, oo), 0(p) x O(oo)) (00 + oo), 00) x 0(oo)) 

(Sp(p, oo), SpO) x Sp(oo)) (SpO + oo), SpO) x Sp(oo)) 

In pairs (G, K) of the left column, the group G ranges in all irreducible 
infinite- dimensional classical groups^. In the first table in the first column, the 
subgroup K is a maximal heavy subgroup in G. The maximal heavy subgroup 
in UO, oo) is U(oo). 

Sphericity theorem. All the above listed pairs (G, K) are spherical. 

Spherical representations are classified, see [31], [24], [25], [28], spherical 
functions are given by simple explicit expressions. For instance, consider the 
pair (GL(oo,R), 0(oo)). Spherical representations are determined by finite col- 
lections of real parameters, s\, . . . , s p and also a £ M , a = 0, 1. Spherical 
functions are given by 

1 • \ -i/2 

ISk A , 1-iSk 

; A, 



*., a ,*(g) = I det gr ■ sgn • (det g)-JJ JJ ( ^-^A, + 



where Xj denote the singular numbers of g E GL(oo,R). 

Remark. For the pair (U(oo) x U(oo), U(oo)) a substantial harmonic anal- 
ysis now is constructed, see pQ, apparently it exists for other pairs of the second 
column, see [TB]. □ 

The class of spherical representations is not closed with respect to natural 
operations (tensor products, restriction to subgroups), it is reasonable to extend 
it. 



3 Of course, the groups U(co), O(oo), Sp(oo) also are classical, p = 0. 
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1.5. Admissible representations. Fix a pair (G,K) from the list. We 
say that a unitary representation of G admissible ([25]), if it is continuous on 
the heavy subgroup K. We also use as a synonym the term " representations 
of the pair (G, if)" . Certainly it is possible to define admissibility in terms of 
appropriate topologies on G (see a discussion in [15]). 

Lemma 1.2 Spherical representations are admissible. 

Proof. It is easy to show that the subspace U a H^ is invariant with respect 
to G. On the other hand it is non-zero (it contains the spherical vector). In 
virtue of irreducibility, it is dense. □ 

We emphasis that groups of finite matrices GL(oo,R), U(oo), O(oo, C) etc. 
are not type@ I groups, and theory of their unitary representations in the usual 
sense is impossible. The class of admissible representations is observable and 
substantial works on representations of classical infinite-dimensional groups usu- 
ally can be included to this scheme. 

Remark. The list contains pairs (0(2oo),0(oo) xO(oo)), (0(2oo),U(oo)). 
The group G in both cases is the same, but conditions of continuity with respect 
to K are different. Therefore we get two different classes of representations. □ 

Remark. The list contains the pair (U(oo) x U(oo),U(oo)). Its represen- 
tations are not tensor products p\ Cg> pi of representations of U(oo) (the corre- 
sponding finite-dimensional theorem, see [5], 13.11.8, in this case is not valid), 
on representations of this pair, see [3D], [3T], [35] ■ □ 

1.6. Product of double cosets and train. Let p be an admissible 
representation of the pair (G, if) in the space H. As above, let be spaces 
of if "-fixed vectors. Denote by the operator of orthogonal projection to 
ffH. We define operators 

It easy to verify that for any hi S if", hi <E if 3 we have 

PaA h2 9 h i) = Pa,p(s)- 

Therefore the function ~p a a is defined on double cosets rj e \ G/K a . 

Multiplicativity theorem. There exists a natural multiplication (g, f)) h-> 

K^\G/K P x K.P \ G/K a -> if 7 \G/if", 

defined for all a, f3, 7 € Z+. Moreover, for any admissible unitary representa- 
tion p of the pair (G, K) the following equality holds 

P/3,-y(s)Pa,p(h) =Pa, 7 (fl° (>)• ( L1 ) 



4 For definition, see, e.g., [5]. 
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We get a category, we call it by train %{G,K) of the pair (G,K). Objects 
of the train are indices (multi-indices) a 0, and morphisms are double cosets. 
The formula (|l.ip claims that we get a representation of a category (see [15] , 
II. 5), i.e., a functor from T(G, i^) to the category of Hilbert spaces and linear 
bounded operators. 

The operation g i-> g- 1 induces the map K a \ G/K fj K fj \ G/K a , we 
denote it by g i-)- g*. It satisfies the identity 

(g°j))* = r°fl*. 

The representation p is a * -representation in the following sense: 

p(&*) = p(q)- 

Approximation property. Let r be a representation of the category T(G, K) . 
Let ||t(0)|| ^ 1 for all q, t(1) = 1. Then the representation r has the form ~p 
for uniquely defined representation p of the pair (G, K) . 

1.7. Comparison with finite-dimensional case. Let G be a Lie group, 
L a compact subgroup, not necessarily maximal. Denote by M.(L \ G/L) the 
space of finite measures that are invariant with respect to left and right shifts 
by elements of L. Evidently, M.(L \ G/L) is an algebra with respect to the 
convolution. For instance, if G is a p-adic GL(n) and L iwahoric subgroup, 
we get the Hecke algebra (see. [HJ); if G is a real Lie group of rank 1 and 
L the maximal compact subgroup, then we get the hypergroup of generalized 
translate, see [9]. 

Let p be a unitary representation of G in the space H . Denote by H L the 
space of L-fixed vectors. For p £ Ai(L\G/L) the operator p(p) has the block 

form f^Q^ with respect to the decomposition H = H L © (H 1 ")- 1 . Thus 

we get a functor from unitary representations of G to representations of the 
convolution algebra M.[L \ G/L). 

It turns out that for infinite-dimensional groups the convolution of double 
cosets can degenerate to a multiplication. A first example, apparently was dis- 
covered by R.S.Ismagilov [6], [7], further [21], [23], [25], [H], numerous examples 
are discussed in [IB] . 

1.8. Explicit formula for the product. See [25]. For definiteness, 
consider the pair (G,K) = (GL(oo, R), O(oo)) and double cosets \ G/K a . 

In other words, we consider matrices g — [ ^ ^ ] of size (/? + oo) x (a + cx)) 



up to the equivalence 



A B\ (1 0\ (A B\ fl a 
G D ~ I U \C D \ V 



(1.2) 



where U, V are orthogonal matrices, l a denotes the unit matrix of size a. The 
product in the category T is given by 

'A B 0\ (P Q\ ( AP B AQ S 











(c 


Dj 


KS 





:= [ G 13 1 = GP Z3 GQ | (1.3) 
1/ U T/ V i? T 
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Note that a similar (but not precisely the same) algebraic structure is known 
as a product of operator colligations, see, e.g., [3]. 

1.9. Characteristic functions (below we do not develop this topic). It 
appears (see [15], IX. 4), that o-multiplication can be transformed to a more 
customary operation^ in the following way. Fix A e CUoo. Write the equation 





( 


Xx 




P- 




\xj 


\ 



A B 
C D 







A B 
C D 



\ 


(q+\ 




y 




q- 


) 


W 



Consider all vectors (p + ,p-) © (q + ,q-) 6 C 2 ^ © C 2q , for which this equation 
has a solution as an equation with indeterminates x, y. We get a subspace x(A) 
of half-dimension in C 2f3 © C 2q depending on A. We regard it as an relation (a 
multi- valued map) from C 2a to C 2/3 . 

On this language, o-multiplication corresponds to the product of character- 
istic functions, i.e., pointwise product of relations. 

Note that xW i s a rational map from Riemann sphere to the Lagrangian 
Grassmannian in C 2 " ffi C 2/3 . The function xW 1S al so ./-contractive in the 
Potapov sense [25] . 

1.10. Self-similarity. See. [26]. Let (G,K) = (GL(oo,R),0(oo)). Denote 
by G a C G the stabilizer of initial a basis vectors in £ 2 - Then the pair (G a , K a ) 
is isomorphic to (G, K). Similar pairs are defined in all the cases, but a definition 
requires some care: we take basis vectors fixed by K a , then G a is their stabilizer 
in G. 

Fix an unitary representation p of (G, K ) . Its restriction to a sufficiently 
small subgroup (G a ,K a ) ~ (G,K) contains a spherical subrepresentation. All 
spherical representations of (G, K) obtained in this way are equivalent {for all 
a). 

1.11. Mantle. Let (G,K) = (GL(oo, R), O(oc)). Split N into two count- 
able subsets N = S U f2 (for instance, odd and even numbers). Denote by 
G~ C G (respectively G(H)) the stabilizer of basis vectors enumerated by ele- 
ments of S (respectively, ft). In other words, G a and G(S) consist of matrices of 



the forms 



and 
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, where blocks correspond to the decomposition 



1 0^ 

v gj \0 1, 

£ 2 (N) = £ 2 (S) ®£ 2 (fl). Evidently, (G(E),K(E)) is isomorphic to (G, K), We 
fix the isomorphism 



i T :(G,K)^(G(E),K(E)), 
induced by some bijection r : N — > H. 



5 This is an analog of Livshits characteristic function, which is a tool in spectral theory 
of non-self-adjoint operators, see |29| . [3]. The Livshits function also take part in the 

representation theory of the pair (U(l + oo) X U(oo), U(oo)), see. |27| . 
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Let p be a unitary representation of (G, K) in the space H. Denote by H^ a ' 
the space of K B -fixed vectors. By we denote the operator of projection on 

Lemma about self-restriction. The space is G(S) -invariant. The 
representation p o i T of the pair (G, K) in is equivalent to p. 

On the other hand, for each g e G we assign the operator p s s (g) : — >■ 
as 

P S , s (g) = P [s] p(g). 
As above, ~p is a function on double cosets. 

Version of multiplicativity theorem. Double cosets Too = K a \ GjK" 
admit a natural structure of a semigroup (mantle of the pair (G,K)), the map 

3 is a representation of Too ■ 

Product is given by the same formula (|1.3j) . in the present situation all blocks 
of the matrix are infinite. 

Next, notice that the natural map G(S) ~ G — > K~ \ G/K~ is injective, 
therefore we observe that the representation of the semigroup K a \ G/K~ ex- 
tends the representation p of G(S) ~ G. 

As a result, we get 

Theorem about mantle. Any unitary representation of the pair (G, K) 
can be extended canonically to a representation of the semigroup Too- 

1.12. Spherical characters. Notice, that Z := K~\G 3 /K 3 C K~\G/K 3 
is a central subsemigroup in the mantle. In any irreducible representation p of 
the pair (G, K) it acts by scalar operators. This determines a multiplicative 
character $ p : Z ->• C x . 

If p is spherical, then $ p coincides with the spherical function of p. Generally, 
it coincides with spherical functions of subgroups as in 11.101 

1.13. Non-standard pairs. Until this moment we discussed pairs (G,K) 
containing in the Olshanski list. Now let G = GL(oo,IR) x • • • x GL(oo,IR), let 
K C G be the group O(cx)) embedded to G by the diagonal. 

a) The pair (G, K) is spherical. Notice, that this statement has no finite- 
dimensional analogs. Multiplicativity theorem also holds. 

b) Nessonov [19] , [20] obtained description of all spherical representations of 
this pair. 

c) The paper [18^ contains a construction of characteristic functions, it is 
valid in our case. 

There arises the following question: In which generality the statements of 
Subsections [L4HTT21 hold? 

1.14. Purposes of the paper. We introduce class of (G, L)-pairs, where 
G is a classical group, L is a heavy subgroup. For this class the following hold 
true: 

— the multiplicativity theorem; 

— the approximation property; 
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— the construction of the mantle; 

— existence of spherical characters. 

The construction of characteristic functions given in [IB] survives in these 
cases, (in the present paper this topic is not discussed). 

Also, we define a subclass of pure (G, L) -pairs. In this case there hold 

— sphericity theorem; 

— self-similarity. 

Definition of (G, L)-pairs is contained in Section 2, sketches of proofs in 
Section 3, Section 4 contains conjectures and additional remarks. We do not 
repeate formulations of theorems. 



2 Definition 



2.1. Straight embeddings of heavy groups.. First, let heavy groups L, K 
be irreducible, i.e, they have the form U(oo), O(oo), Sp(oo). There are obvious 
homomorphisms : 



O(oo) -> U(oc) 
U(oo) -4 0(2oo) 



O(oo) -> Sp(oo) 
Sp(oo) -> O(4oo) 



U(oo) -> Sp(oo) 
Sp(oo) -4 U(2oo). 



Also element-wise complex conjugation determines a homomorphism U(oo) to 
itself. We call such homomorphisms and also identical homomorphisms g n> g 
trivial. 

Next, let K be irreducible, L = L\ x • • ■ x L p . We say that a homomorphism 
L — > K is straight, if it has the form 



(#1, •••,&>) 



/la 

n(5mj 







5fc e Lfc, 



(2.1) 



TN{g mN )J 



where Tj are trivial homomorphisms. We admit unit matrix l a of arbitrary 
order 0, 1, 2, ... , oo. The number N can be in the limits < N < oo. We do 
not require a homomorphism to be faithful. 

Finally, let L = L\ x • • ■ x L p , K = K% x • • • x K q . Then a straight homo- 
morphism is a product of straight homomorphisms L — > K\ -ftT . 

2.2. (G, i)-pairs. We say that a classical group G = G\ x • • • x G r is a finite 
product of irreducible classical groups (see the list above). Let K = K\ x • • • x K q 
be the maximal heavy subgroup in G. Let L be a straightly embedded heavy 
subgroup in K. We call such objects {G,L) -pairs. 

A pair (G, L) is pure, if the centralizer of the subgroup L in G is triviajf). 



6 Examples. The pair (G,L) = (GL(3oo, R), O(oo) X O(oo) X O(oo)) is pure. The centralizer 
of L in the group of all bounded operators is (R x ) 3 . But it is not contained in the group 
G = GL(3oo,R) of finite matrices. The pair (0(1 + 3oo,R), O(oo) X O(oo) X O(oo)) is not 
pure, the centralizer in 0(1 + 3oo,R) is Z2. 
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An equivalent condition: in matrices (|2.1[) the unit block 1 Q is absent and there 
are no groups U(p, oo), 0(p, oo), Sp(p, oo) among factors GjQ 

We say that a pair is finite, if blocks 1 Q in formula (|2.1[) for all factors .K; 
have hnite sizes. Finite pairs will appear only in conjectures in the last section. 

2.3. Definition of multiplication on double cosets. Let L be irre- 
ducible. Fix a ^ and consider the sequence 6m ei given by 



/la 
















1 m 
















\o 








loo/ 



(2.2) 



If L = L\ X • • • X L p is reducible, we fix an multi-index a and take the net 
depending on a multi-index m, 

Let 

e A' H \ G/id* 31 , t) e KW \ G/K [a] . 

Choose representatives g € JJ, h 6 f). 
Consider subsequence 

Proposition 2.1 a) vl double coset i m — K^z m K^ is eventually constant. 

b) The result g o h = limjm does not depend on a choice of representatives 

g e s, h e fj. 

c) T/ie operation obtained in this way is associative, i.e., for each a, f3, 7, S 
and any 

feK s \ G/m, fl G K"> \ G/K 13 , t)€KP\ G/K a 

we have (f o g) o |j = f o (g o f)). 

As a result we get a category, we call it train of the pair (G, L) and denote 
by 1(G,L). 

Instead of formal proof in general case, we present examples explaining why 
this happens. 

2.4. Example. Consider the pair (GL(oo, R), O(oo)) . To simplify notation, 
assume 7 = /3 = a. Fix g, h € G. Write g, /i as block matrices of the size 
a + N + 00. Let N be sufficiently large. Then g, h have the form 



(2.3) 







b 




p 


9 


S 






d 


l ) 


h = \r 


t 











loo/ 







loo/ 



7 This limitation is necessary, because for such groups the maximal heavy subgroup is not 
a precise analog of the maximal compact group. 
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The product gQ^ +k h equals 



/ ap 


aq 





h 







cp 


cq 





d 




















lk 





r 


t 




















1a. 











\° 














loo/ 



Since we examine double cosets, we transpose rows and columns whose numbers 
> a. In this way we can reduce our matrix to arbitrary of two convenient forms: 



Si 



I ap aq b \ 

cp cq d 

r t 

\0 l 2k +oo) 



So 



I ap b aq \ 

cp d cq 

r t 

\o l 2fe+00 / 



We observe that the result is independent on k. Next, we must verify indepen- 
dence on a choice of representatives, i.e., gu, vh c u, v £ K a produce the same 
double coset. Without loss of generality, we can assume u, v € O(N) (otherwise 
we choose larger N earlier). We replace g, h from (|2.3|) to 



(2.4) 















<7 












*- 


vr 
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with orthogonal matrices u, u. The new matrix S2 is 

A 



/ ap few aq 0\ 

cp du eg 

wr ut 

\ 1/ 



1 



V 



V 



^ /ap 6 ag 0\ A 

cp d cq 

r f 

\0 1/ 



V 



V 



Associativity is clear from the formula (|1.3p . 

2.5. Two examples, a) Consider the pair G = GL(oo, E) x • • • x GL(oo, R) 
with the diagonal subgroup L — O(oo). We have a finite collection of matrices 
a,- bj 



defined up to the equivalence 



ai 61 
ci di 

lp 
u 



ai h 
ci di 



1/3 

u 



Then o-product is defined component-wise by the formula ([1.3 
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b) Let G — U(2oo), K — 0(oo) x 0(oo). Now we have unitary matrices g 
determined up to the equivalence 



/ an 

Cll 
021 
\c 2 l 



da 
l>2i 
d>i 



0.12 
(-12 
22 
('22 



bi 2 \ 

d.12 
b 22 
d 22 / 



/I 



V 



til 



\ 



112/ 



( an 

Cll 

a2i 

\C21 



&11 

d\\ 

1)21 

(hi 



ai2 

C12 
"22 
C22 



6l2^ 

di2 

&22 



/I 



Vl 



V2j 



where u\, U2, «i, «2 G 0(oo). 
The product is given by 



9°9 



( an 


fell 





ai2 


&12 











b'n 


a'i2 





b'i2\ 


Cll 


(111 





Cl2 


di2 










1 




















1 













c'n 





d' n 


C'l2 





d'12 




1)21 





022 


&22 







a'21 





b'21 


a 22 





b'22 


C21 


d-21 





C22 


^22 



















1 



















V 









d'21 


C 22 








(2.5) 



Double cosets are taken with respect to the group of matrices of the form 
/ I \ 



"11 U12 
U21 u 2 2 



1 



«u V12 
V21 V22J 



where 



Uu U12 

U21 u 2 2 



Vll V12 
V21 V 2 2 



G 0(2oo) 



3 Proofs 

3.1. Multiplicativity (the analog of II. 6[) . 

Lemma 3.1 For any unitary representation it of a heavy group L the sequence 
7r(e^ ] ) (see ^T^) weakly converges to the projector P^ to . 

See |15j . Theorem VIII. 1.4. A priory proof is given for a symmetric group 
S(oo), proofs for 0(oo), U(oo), Sp(oo) are the same. On the other hand, it is 
easy to reduce the statement from the explicit classification of representations 
of heavy groups [23] , see also [15] . □ 

Let us prove the multiplicativity theorem for an arbitrary pair (G,L). Let 
Q, f), <?, h be the same as in Subsection 2.3. Then 

for sufficiently large m. On the other hand, 

lim P^p(geMh) = lim P^ p(g) p{Q^) p{h) = 

m—^oo m— >oo 

= P™p(g)( lim p(e^))p(h) = pWp{g)pW p (h) = p(fl)p(&). □ 

m—¥oo 
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Thus, for any unitary representation of (G, L) we construct a representation 
of the category T(G, L). Now we present the inverse construction. 

3.2. Approximation property. Let a < f3 (therefore, L a D L"). Denote 
by \a : p the double coset ■ 1 • L a . Set fip^ :— A* g. Then the following- 
identities hold: 

f-f3,a ° ^a,P — 1 j ^/3^^a,/3 = A Qj7 , (3.1) 

where 1^ denotes the unit L a ■ 1 ■ L a of the semigroup L a \G/L a . We get a 
structure of ordered category in the sense of |15j . III. 4. 

Notice that ip a ,/3 ■— ^a,/3 ° fJ>p,a as a subset in G coincides with L a . But it 
is a nontrivial idempotent in L 13 \ G/L@: 



Consider a ^-representation R of the train T(G, L) by contractive operators, 
let R assigns unit operators to units of semigroups of endomorphisms. Denote 
by H^ a ' the Hilbert space, corresponding to a (multi)index a. In virtue of (|3.ip , 
an operator R(X a ^) is an operator of isometric embedding — > H^. The 
projection operator to the image of the embedding is ipa,p- Now we have a chain 
of isometric embedding (to be definite, let a = (ai,a 2 ) be a bi-index): 



H 



Oil + 1,CK2 



jjai + l,Q2 + l 



Denote by H the limit space. 

The group G is a union of finite-dimensional groups G(a). Notice that the 
natural map G(a) to L a \ G/L a is an isomorphism. Therefore we get represen- 
tation of G(a) in the space H [ Q l . Operators of representation are contractive 
with their inverses. Therefore they are unitary. 

Hence the group G acts in H . We omit a watching of the remaining details. 

□ 

3.3. Sphericity. 

Lemma 3.2 For a pure (G, L)-pairs the semigroup L \ G/L is commutative. 

The case is that product in such L \ G/L means that we "join" one matrix 
to another. For instance, formula (|2.5j) in this case is reduced to 



g°g = 



(dn 


di 2 \ 


'it 


d[ 2 \ 


\d 2 i 






d'22) 



(d u 





di2 


°\ 





d' u 







d 2 \ 





1^22 





V° 


d'21 





d'22) 
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To get g' o <?, we conjugate right-hand side by a matrix 



J 
J 



where J = 



[0 ljv ^ 

loo 

1 N 

V o o o 



with sufficiently large N. □ 
Thus L\G/L is an Abelian semigroup with involution. Its irreducible repre- 
sentations are one-dimensional. On the other hand, irreducibility of representa- 
tion of (G, L) is equivalent to irreducibility of the corresponding representation 
of the train T(G, L); but for an irreducible representation of an ordered cat- 
egory, all representations of semigroups of endomorphisms are irreducible (sec 
[T5] . Lemma III. 4. 3). This implies also the following statement. 

Proposition 3.3 Spherical functions of the pair (G, L) are homomorphism^ 
from the semigroup L \ G / L to the multiplicative group C x . 

3.4. Self-similarity. Consider a pure pair (G,L) and its subgroup G a as 
in 1.8. 

Lemma 3.4 The image of G a in L a \G/L a is a central semigroup canonically 
isomorphic to L\G / L. 

PROOF. The isomorphism is induced by the "shift" G — > G a , i.e., g n> 

q ^ . Further, let g € G, h € G a . It is easy to see that for sufficiently large 

m, 

g(e^het ] ) = (&t ] h&t ] )9 =* L a - g eMh-L a =L a -hQWg-L a , □ 

Next, for a < /3 there is a natural map K fj \ G/K 13 ->• K a \ G/K a that can 
be defined by the formula 

11:94 jtt^.o. o g o A Qi)3 . 

Lemma 3.5 The projection U : \ G p / -+K a \ G a /K a of central semi- 
groups coincides with isomorphism induced by the shift G 3 — > G a . 

Proof. The element g G G a and the corresponding element g G G^ are 
conjugate in G a by an element of the group K a . □ 

Therefore the semigroup L\G/L acts in the whole space H. If a repre- 
sentation p is irreducible, then this semigroup acts by scalar operators. The 
corresponding character coincides with spherical functions of groups (G a ,K a ) 
acting in H . 

3.5. Self-restriction. Let us prove the statement for (G, L) = (GL(oo, M), O(oo)). 
We preserve the notation of 11.111 



'non-arbitrary. 
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The map r : N — > 5 determines 0-1 matrix, denote it by A, the norm of 
A is 1. According Theorem II. H representations of a heavy group admits a 
weakly continuous extension to representation of semigroup of contructions. In 
particular, for any continuous representation n of L in a Hilbert space H, there 
is an operator ir(A) assigned to the matrix A. 

Lemma 3.6 tt(A) is an isometric map from H to the subspace H^. 

Proof. First, (ir(A))*ir(A) = n(A*A) = tt(1) = 1. Therefore n(A) is an 
isometric embedding. Similarly, tt(A)(tt(A))* = tt(AA*). The operator AA* is 
the projection to the subspace $j(H) in Next, we again refer to explicit 

form of representations of heavy groups, [23]- □ 

Consider an admissible representation p of a pair (G, L). Denote by it the 
restriction of p to L. Extend tt to the semigroup of contractions. 

Lemma 3.7 The operator it (A) intertwines the representation it of the pair 
(G, L) in the space H and the representation n o i T in the space . 

Proof. Consider a sequence of matrices-permutations B m weakly converg- 
ing to A. Then ir(B m ) weakly converges to it {A). The group G is an inductive 
limit of reductive groups G(n). For each group G(n) the operator ir(B m ) with 
sufficiently large number m is intertwining. Therefore the limit is intertwining 
for the whole group G. □ 

Let L — L\ x • • • x L p . Each factor is a group of unitary operators in a 
certain space l\ = ^(N). We split each copy of N into two countable subsets 
N = 3j U ilj. Fix bijections Tj : N — ) 3j. Now we can repeat the same words. 

3.6. Mantle. Theorem on mantle follows from multiplicativity theorem 
and lemma about self-restriction. 

3.7. Spherical characters, see 11.121 Consider a (G, L)-pair, acting in a 
direct sum of the spaces £2 as in 12.1142.21 Consider all basis vectors in ©£2 
that are fixed by the group L. Consider the subgroup G m i n C G stabilizing all 

Example. Let (G, L) = (GL(7 + 2oo, K), O(oo) x O(oo)). Then G mm = 
GL(2oo,R). 

Proposition 3.8 Mantle of the pair (G , L) contains a central semigroup canon- 
ically isomorphic L \ G m i n /L. 

Explain this by an example. Let (G,L) — (U(p + 00) x U(oo), U(oo)) , the 
subgroup L ~ U(oo) is embedded to the product by the diagonal, G m in — 
U(oo) x U(oo). Redenote (G, L) as (U(p + 2oo) x U(2oo), U(2oo)) , we write its 
elements as pairs of block matrices (g,h), where the size of g is (p + 00 + 00) and 
the size of h is (00 + 00). Consider a subgroup Z ~ U(oo) x U(oo) consisting of 
pairs of matrices 
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where <fi, tfe € U(oo). Consider a subgroup M C Z consisting of pairs satisfying 
d\ = d.2- Then the mantle is the semigroup M \ G/M. It contains the central 
subgroup M \ Z/M. Property of centrality is a version of Lemma OH] □ 

4 Discussion 

4.1. General conjectures. Let a pair (G,L) be finite, see 12.21 Then 

a) Unitary representations of (G, L) have type /. 

b) There exists only countable set of representations with a given spherical 
character. 

c) Let (G, L) — > (G 1 , L') be an embedding of pairs such that the embedding 
L — > L' is straight in the sense of 12.11 Then the restriction of any irreducible 
representation p of the pair (G', L') to (G, L) has a pure discrete spectrum. In 
particular, this is the case for tensor products of irreducible representations. 

d) An explicit classification of spherical representations is possible. Precisely, 
they can be obtained by the construction described in [TS] . IX. 5 (the difference 
between symmetric and non-symmetric pairs is explained in IX. 5. 6). 

e) In [18] for any element g £ L a \ G/L 13 there were obtained spectral 
data: "characteristic function" (it is an inner function of a matrix argument) 
and " eigensurface" (a hypersurface in a Grassmannian) . The element q can be 
uniquely reconstructed from these data. 

4.2. Why we consider only such pairs? There are many embeddings of 
U(n) to unitary groups U(iV) of big size. For instance, g i— >• g®g determines an 
embedding U(n) — > U(n 2 ). But for finite g € U(oo) the matrix g®g is not finite. 
I.e., formally, embeddings U(n) to U(-), different from straight embeddings do 
not survive as n — > oo. 

Certainly, this argument is not sufficient. We can search natural subgroups 
in the complete operator group in ^(N x N) containing matrices g ® g- The 
author do not see such subgroups @. 

4.3. Sphericity. There are limits of spherical pairs that are not pure, say 
(U(l + oo) x U(oo),U(oo)), (0(l + 2oo),U(oo)), see [27]. 

Also there are many cases of sphericity with respect to subgroups of the form 
L x M, where L is heavy and M is compact (see, e.g., the pair U(p + oo)/U(p) x 
U(oo) mentioned above). 

4.4. Symmetric groups. For symmetric groups similar facts hold, an 
example is discussed in [17 . Our proofs are based on Lemma 13.11 they survive 
literally for symmetric groups. 

9 For instance, the subgroup generated by finite unitary matrices and matrices g ® g is a 
direct product of two generating subgroups. 
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